Skip to content

test(v2): live contract tests for all 5 client.v2 endpoints#109

Merged
yzld2002 merged 3 commits into
mainfrom
feat/v2-contract-tests
Jul 13, 2026
Merged

test(v2): live contract tests for all 5 client.v2 endpoints#109
yzld2002 merged 3 commits into
mainfrom
feat/v2-contract-tests

Conversation

@yzld2002

Copy link
Copy Markdown
Member

Adds live contract tests for all 5 client.v2 endpointsfiles.upload, extract (sync), extract_jobs (create+wait), parse (sync), parse_jobs (create+wait) — in tests/contract/test_v2_smoke.py (marker contract), plus a small PDF fixture (tests/contract/sample.pdf) for the parse checks.

How they run

  • They plug into the existing pr-gates.yml contract-tests job (rye run pytest tests/contract -m contract -n 0) — no workflow change. That job is scoped to spec-sync/* PRs and skips cleanly without LANDINGAI_ADE_STAGING_APIKEY (forks/local), so ordinary PRs (like this one) are not gated on live staging.
  • Client uses environment="staging" → V2 routes to api.ade.staging.landing.ai. Assertions are structural (types/shape/JobStatus.COMPLETED), not exact-value, since it's a live gateway. wait() calls are bounded (timeout=300).
  • Verified: skips cleanly without the key (6 skipped: 1 V1 + 5 V2); ./scripts/lint clean (ruff + pyright-strict + mypy).

⚠️ Live-verification deferred (staging outage)

As of 2026-07-12, all *.staging.landing.ai hosts return 404 on every path (including the openapi.json endpoints that served 200 on 2026-07-08) — an aide/infra-side change, not a test defect. The pre-existing test_v1_smoke.py and examples/v2_smoke_test.py fail identically, confirming it's environment-level. The same V2 code passed 5/5 live against staging on 2026-07-08, so these tests are correct by construction; a fresh green run is pending staging recovery. (Separately noted: prod V2 is now liveapi.ade.landing.ai/v2/parse → 401, prod spec aide.landing.ai/openapi.json → 200.)

Related: #95 (V2 SDK), the spec-sync contract-tests job in pr-gates.yml.

🤖 Generated with Claude Code

Add tests/contract/test_v2_smoke.py covering files.upload, extract (sync),
extract_jobs (create+wait), parse (sync), and parse_jobs (create+wait)
against live staging, mirroring test_v1_smoke.py conventions.
Copilot AI review requested due to automatic review settings July 12, 2026 07:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds live staging contract coverage for all five synchronous and job-based V2 client surfaces.

Changes:

  • Adds five V2 contract smoke tests.
  • Adds a PDF fixture for parse tests.
  • Uses bounded polling and skips when staging credentials are unavailable.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.

File Description
tests/contract/test_v2_smoke.py Exercises V2 upload, extract, and parse APIs.
tests/contract/sample.pdf Provides parse-test input.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/contract/test_v2_smoke.py
@yzld2002 yzld2002 added the breaking-change-approved Surface-lock override: a reviewed, intentional public-surface change label Jul 12, 2026
Copilot AI review requested due to automatic review settings July 12, 2026 07:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated no new comments.

…hanges

griffe flags a changed value of a public constant (e.g. adding staging/dev to the
ENVIRONMENTS map) as breaking, tripping the gate on every PR since #105. Additive
config is not an API-signature break, so filter griffe's 'Attribute value was changed'
breakages; any other breakage still fails the gate (verified: renaming a public method
still trips it). Removes the need for the breaking-change-approved override on such PRs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@yzld2002 yzld2002 removed the breaking-change-approved Surface-lock override: a reviewed, intentional public-surface change label Jul 13, 2026
Copilot AI review requested due to automatic review settings July 13, 2026 03:28
@yzld2002

Copy link
Copy Markdown
Member Author

Update: live-verified + surface-lock gate fix folded in

Live verification (staging is back): ran the full contract suite against staging — 6/6 pass (1 V1 + 5 V2: files.upload, extract sync, extract_jobs, parse sync, parse_jobs incl. the new result-payload assertion). The deferred live check is now green.

surface-lock gate fix (293fa2e): per request, this PR now also fixes the gate so it stops tripping every PR on the benign additive ENVIRONMENTS value change. surface-lock.sh now ignores griffe's "Attribute value was changed" breakages (additive public-constant config isn't an API-signature break); any other breakage still fails it (verified: renaming a public method still trips the gate). Removed the breaking-change-approved override label so this PR's CI exercises the fixed gate directly.

Follow-up (not this PR): surface-lock doesn't currently track the V2 resource-method signatures (files.upload, parse_jobs.create, etc.) because those classes aren't exported from a public __init__ (only V2Resource/AsyncV2Resource are) — griffe's static traversal doesn't reach them. The live contract tests cover them behaviorally; tightening surface-lock's static coverage of client.v2.* is a separate improvement.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum diff size. Try reducing the number of changed files and lines, and requesting a review from Copilot again.

@yzld2002

Copy link
Copy Markdown
Member Author

Correction to the surface-lock coverage note above

My earlier note ("surface-lock doesn't track V2 resource-method signatures because they aren't exported from a public __init__") was wrong. Verified: griffe does statically track the V2 resource methods — renaming FilesResource.upload and diffing against a V2-containing ref reports FilesResource.upload: Public object was removed.

The actual reason V2 isn't covered yet is the baseline: surface-lock diffs against the last release tag (v1.12.0), which predates V2, so the entire client.v2 subtree is "new" relative to it and can't register as a breaking change. This resolves automatically at the first release that includes V2 — from then on the baseline tag contains client.v2, and its method signatures are locked for free. No code change is needed (and exporting the classes wouldn't help). Until that release, V2 is covered behaviorally by the live contract tests added in this PR.

@yzld2002 yzld2002 merged commit e80bcfe into main Jul 13, 2026
5 of 6 checks passed
@yzld2002 yzld2002 deleted the feat/v2-contract-tests branch July 13, 2026 06:43
yzld2002 added a commit that referenced this pull request Jul 13, 2026
…ng loop

Add a parallel `spec-sync-v2` job that mirrors the proven V1 loop, pointed at the
V2 spec on the AIDE gateway (aide.staging.landing.ai/openapi.json) and a separate
`spec-sync/v2` branch, with a V2-tailored wiring prompt. Reuses the existing
check-drift / fetch-normalize / gen-models scripts unchanged; the V1 job is
untouched.

The committed baseline `specs/v2-aide.json` is the FULL current spec, so
check-drift reports no drift and the loop ships live but quiet — it fires only on a
future real V2 spec change. /v2/workflow is intentionally deferred (not shipping
yet): it stays in the baseline (so it isn't flagged as drift) and the AI-wiring
prompt explicitly excludes the three /v2/workflow* routes so a future unrelated
drift can't pull them in.

Hosts (do not conflate): V2 spec = aide.[env]; V2 API = api.ade.[env]. The V2
contract tests from #109 gate the spec-sync/v2 branch as the live guardrail.

Design: docs/design/v2-spec-sync-extension.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
yzld2002 added a commit that referenced this pull request Jul 13, 2026
…ant AI-wiring) (#110)

* feat(spec-sync): track V2 spec drift (AIDE gateway) + dormant AI-wiring loop

Add a parallel `spec-sync-v2` job that mirrors the proven V1 loop, pointed at the
V2 spec on the AIDE gateway (aide.staging.landing.ai/openapi.json) and a separate
`spec-sync/v2` branch, with a V2-tailored wiring prompt. Reuses the existing
check-drift / fetch-normalize / gen-models scripts unchanged; the V1 job is
untouched.

The committed baseline `specs/v2-aide.json` is the FULL current spec, so
check-drift reports no drift and the loop ships live but quiet — it fires only on a
future real V2 spec change. /v2/workflow is intentionally deferred (not shipping
yet): it stays in the baseline (so it isn't flagged as drift) and the AI-wiring
prompt explicitly excludes the three /v2/workflow* routes so a future unrelated
drift can't pull them in.

Hosts (do not conflate): V2 spec = aide.[env]; V2 API = api.ade.[env]. The V2
contract tests from #109 gate the spec-sync/v2 branch as the live guardrail.

Design: docs/design/v2-spec-sync-extension.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(spec-sync): address Copilot review on the V2 loop

- Security: drop rye/scripts shell from the AI step's allowedTools (only
  Edit/Write/Read/Glob/Grep/git-diff) so a prompt-injected spec can't execute
  code with the push PAT; add a fixed step that discards any agent edit to
  scripts/.github/specs before running the trusted ./scripts/format.
- Broaden the wiring prompt to cover every changed operation backing client.v2
  (field changes on existing ops + /v1/files, not only new /v2 routes).
- Permit backward-compatible signature additions (new optional kwargs); verified
  surface-lock does not flag an added optional parameter.
- Docs: resources/v2/__init__.py stays unchanged (no sub-resource re-export);
  genericize the Phase-2 design away from the deferred workflow feature.

Follow-up (V1 + V2): persist-credentials:false + read-only action token.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(spec-sync): harden V2 loop per second Copilot round

- Sanitizer: enforce a product-code ALLOWLIST (src/tests/docs/api.md) instead of
  a 3-dir deny-list, before format AND staging. A repo-wide agent Write to e.g.
  pyproject.toml (which defines `rye run format`) can no longer execute in the
  trusted format step or reach the commit via `git add -A`. Verified in a scratch
  repo: out-of-scope tracked edits reverted, untracked injections deleted,
  legitimate src/tests/docs additions kept.
- Open-PR check: fail CLOSED — capture `gh pr list` result outside the `if`
  condition so a transient API/auth failure aborts instead of silently becoming
  open=false and force-pushing over an existing reviewed PR branch.
- PR body: mark the AI commit conditional (workflow-only drift / AI no-op produces
  a mechanical-only PR), so mechanical-only PRs don't carry an inaccurate summary.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(spec-sync): wire component-only V2 drift (response-model fields)

Broaden the AI-wiring prompt to inspect changed components.schemas, not just
operation/path hunks: a response-field addition changes only a component (e.g.
ParseResponse) and its generated model, not the operation. Instruct the agent to
trace $ref consumers of changed schemas to non-workflow operations, and to
distinguish request fields (new optional method param) from response-model fields
(add to the response type under types/v2/).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants